{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63ae56b0",
   "metadata": {},
   "source": [
    "![](https://www.nuplan.org/static/media/nuPlan_final.3fde7586.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e01e5d2",
   "metadata": {},
   "source": [
    "### Contents\n",
    "\n",
    "1. [Introduction](#introduction)\n",
    "2. [Accessing sensor data through NuPlanScenario](#nuplan_scenario)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34460db1",
   "metadata": {},
   "source": [
    "# Introduction <a name=\"introduction\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01b6b6ee",
   "metadata": {},
   "source": [
    "Welcome to the nuPlan sensor data tutorial. This tutorial is a short guide on how to access sensor data. The sensor data are stored as object blobs (.jpg for images and .pcd for pointcloud) which are separate from the log database (.db) files. The database files simply provide a reference to the corresponding sensor blob. **Hence, you need to have both the database files and the sensor blobs downloaded and stored in the correct directories**. It is recommended that you set up your file structure as illustrated below.\n",
    "\n",
    "Note that the name of the DB corresponds to the directory name where the sensor blob is stored. For example, the sensor blobs for `nuplan-v1.1/splits/mini/2021.05.12.22.00.38_veh-35_01008_01518.db` lives in `nuplan-v1.1/sensor_blobs/2021.05.12.22.00.38_veh-35_01008_01518/`.\n",
    "\n",
    "\n",
    "```\n",
    "~/nuplan\n",
    "├── exp\n",
    "│   └── ${USER}\n",
    "│       ├── cache\n",
    "│       │   └── <cached_tokens>\n",
    "│       └── exp\n",
    "│           └── my_nuplan_experiment\n",
    "└── dataset\n",
    "    ├── maps\n",
    "    │   ├── nuplan-maps-v1.0.json\n",
    "    │   ├── sg-one-north\n",
    "    │   │   └── 9.17.1964\n",
    "    │   │       └── map.gpkg\n",
    "    │   ├── us-ma-boston\n",
    "    │   │   └── 9.12.1817\n",
    "    │   │       └── map.gpkg\n",
    "    │   ├── us-nv-las-vegas-strip\n",
    "    │   │   └── 9.15.1915\n",
    "    │   │       └── map.gpkg\n",
    "    │   └── us-pa-pittsburgh-hazelwood\n",
    "    │       └── 9.17.1937\n",
    "    │           └── map.gpkg\n",
    "    └── nuplan-v1.1\n",
    "        ├── splits\n",
    "        │     ├── mini\n",
    "        │     │    ├── 2021.05.12.22.00.38_veh-35_01008_01518.db\n",
    "        │     │    ├── 2021.06.09.17.23.18_veh-38_00773_01140.db\n",
    "        │     │    ├── ...\n",
    "        │     │    └── 2021.10.11.08.31.07_veh-50_01750_01948.db\n",
    "        │     └── trainval\n",
    "        │          ├── 2021.05.12.22.00.38_veh-35_01008_01518.db\n",
    "        │          ├── 2021.06.09.17.23.18_veh-38_00773_01140.db\n",
    "        │          ├── ...\n",
    "        │          └── 2021.10.11.08.31.07_veh-50_01750_01948.db\n",
    "        └── sensor_blobs\n",
    "              ├── 2021.05.12.22.00.38_veh-35_01008_01518\n",
    "              │    ├── CAM_F0\n",
    "              │    │     ├── c082c104b7ac5a71.jpg\n",
    "              │    │     ├── af380db4b4ca5d63.jpg\n",
    "              │    │     ├── ...\n",
    "              │    │     └── 2270fccfb44858b3.jpg\n",
    "              │    ├── CAM_B0\n",
    "              │    ├── CAM_L0\n",
    "              │    ├── CAM_L1\n",
    "              │    ├── CAM_L2\n",
    "              │    ├── CAM_R0\n",
    "              │    ├── CAM_R1\n",
    "              │    ├── CAM_R2\n",
    "              │    └──MergedPointCloud\n",
    "              │         ├── 03fafcf2c0865668.pcd\n",
    "              │         ├── 5aee37ce29665f1b.pcd\n",
    "              │         ├── ...\n",
    "              │         └── 5fe65ef6a97f5caf.pcd\n",
    "              │\n",
    "              ├── 2021.06.09.17.23.18_veh-38_00773_01140\n",
    "              ├── ...\n",
    "              └── 2021.10.11.08.31.07_veh-50_01750_01948\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0189156",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "To be able to access all resources within this notebook, make sure Jupyter is launched at the root of this repo. The path of the notebook should be `/notebook/<repo_root>`.\n",
    "\n",
    "### IMPORTANT\n",
    "Please make sure that you have the following downloaded:\n",
    "- nuplan_mini dataset from `nuplan-v1.1_mini.zip`\n",
    "- sensor blobs from `nuplan-v1.1_mini_camera_0.zip` and `nuplan-v1.1_mini_lidar_0.zip`\n",
    "\n",
    "Make sure that the files are extracted following the file structure shown above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "779b0132",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Useful imports\n",
    "import os\n",
    "from pathlib import Path\n",
    "import tempfile\n",
    "import hydra\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline\n",
    "\n",
    "from tutorials.utils.tutorial_utils import setup_notebook\n",
    "\n",
    "setup_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edf9156f",
   "metadata": {},
   "source": [
    "# Accessing sensor data through NuPlanScenario<a name=\"nuplan_scenario\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b21f4f69",
   "metadata": {},
   "source": [
    "[NuplanScenario](https://github.com/motional/nuplan-devkit/blob/master/nuplan/planning/scenario_builder/nuplan_db/nuplan_scenario.py#L45) is the primary method of accessing the sensor data. NuPlanScenario provides convenience methods such as [`get_sensors_at_iteration()`](https://github.com/motional/nuplan-devkit/blob/master/nuplan/planning/scenario_builder/nuplan_db/nuplan_scenario.py#L293) and [`get_past_sensors()`](https://github.com/motional/nuplan-devkit/blob/master/nuplan/planning/scenario_builder/nuplan_db/nuplan_scenario.py#L372). These methods return a single [Sensors](https://github.com/motional/nuplan-devkit/blob/master/nuplan/planning/simulation/observation/observation_type.py#L51) dataclass instance or a list of `Sensors` instances, respectively. The dataclass instance contains both lidar pointclouds and images. It is in the form of a dictionary that maps sensor channels to sensor data.\n",
    "```\n",
    "@dataclass\n",
    "class Sensors(Observation):\n",
    "    \"\"\"\n",
    "    Output of sensors, e.g. images and pointclouds.\n",
    "    \"\"\"\n",
    "\n",
    "    pointcloud: Optional[Dict[LidarChannel, LidarPointCloud]]\n",
    "    images: Optional[Dict[CameraChannel, Image]]\n",
    "```\n",
    "\n",
    "Let's see how it works below. First we need to create an instance of NuPlanScenario. Usually, this is taken care of by a builder during training or simulation. However, for the sake of demonstration we will manually create one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8f1b203",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from nuplan.common.actor_state.vehicle_parameters import get_pacifica_parameters\n",
    "from nuplan.planning.scenario_builder.nuplan_db.nuplan_scenario import NuPlanScenario, CameraChannel, LidarChannel\n",
    "from nuplan.planning.scenario_builder.nuplan_db.nuplan_scenario_utils import ScenarioExtractionInfo\n",
    "\n",
    "NUPLAN_DATA_ROOT = \"/data/sets/nuplan/\"\n",
    "NUPLAN_MAP_VERSION = \"nuplan-maps-v1.0\"\n",
    "NUPLAN_MAPS_ROOT = \"/data/sets/nuplan/maps\"\n",
    "NUPLAN_SENSOR_ROOT = f\"{NUPLAN_DATA_ROOT}/nuplan-v1.1/sensor_blobs\"\n",
    "TEST_DB_FILE = f\"{NUPLAN_DATA_ROOT}/nuplan-v1.1/splits/mini/2021.05.12.22.00.38_veh-35_01008_01518.db\"\n",
    "MAP_NAME = \"us-nv-las-vegas\"\n",
    "TEST_INITIAL_LIDAR_PC = \"58ccd3df9eab54a3\"\n",
    "TEST_INITIAL_TIMESTAMP = 1620858198150622\n",
    "\n",
    "scenario = NuPlanScenario(\n",
    "            data_root=f\"{NUPLAN_DATA_ROOT}/nuplan-v1.1/splits/mini\",\n",
    "            log_file_load_path=TEST_DB_FILE,\n",
    "            initial_lidar_token=TEST_INITIAL_LIDAR_PC,\n",
    "            initial_lidar_timestamp=TEST_INITIAL_TIMESTAMP,\n",
    "            scenario_type=\"scenario_type\",\n",
    "            map_root=NUPLAN_MAPS_ROOT,\n",
    "            map_version=NUPLAN_MAP_VERSION,\n",
    "            map_name=MAP_NAME,\n",
    "            scenario_extraction_info=ScenarioExtractionInfo(\n",
    "                scenario_name=\"scenario_name\", scenario_duration=20, extraction_offset=1, subsample_ratio=0.5\n",
    "            ),\n",
    "            ego_vehicle_parameters=get_pacifica_parameters(),\n",
    "            sensor_root=NUPLAN_DATA_ROOT+\"/nuplan-v1.1/sensor_blobs\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1945b81c",
   "metadata": {},
   "source": [
    "Now that we have a NuplanScenario instance we can start using the interfaces available. In this tutorial we will use `get_sensors_at_iteration()`. First we want to extract a lidar point cloud and an image from the front facing camera. Take note of the channels `CameraChannel.CAM_F0` and `LidarChannel.MERGED_PC` passed to the method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11b08c6d",
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "sensors = scenario.get_sensors_at_iteration(0, [CameraChannel.CAM_F0, LidarChannel.MERGED_PC])\n",
    "print(sensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac5032f9",
   "metadata": {},
   "source": [
    "As you can see from the print statement, `sensors` stores two dictionaries. One for pointclouds, and one for images. Each dictionary uses the enums `LidarChannel` and `CameraChannel` to map the sensor data to the correct channels. In nuPlan the available channels are: \n",
    "\n",
    "```\n",
    "class CameraChannel(Enum):\n",
    "    \"\"\"\n",
    "    An enum class representing supported camera channels\n",
    "    \"\"\"\n",
    "\n",
    "    CAM_F0 = \"CAM_F0\"\n",
    "    CAM_B0 = \"CAM_B0\"\n",
    "    CAM_L0 = \"CAM_L0\"\n",
    "    CAM_L1 = \"CAM_L1\"\n",
    "    CAM_L2 = \"CAM_L2\"\n",
    "    CAM_R0 = \"CAM_R0\"\n",
    "    CAM_R1 = \"CAM_R1\"\n",
    "    CAM_R2 = \"CAM_R2\"\n",
    "\n",
    "\n",
    "class LidarChannel(Enum):\n",
    "    \"\"\"\n",
    "    An enum class representing supported lidar channels\n",
    "    \"\"\"\n",
    "\n",
    "    MERGED_PC = \"MergedPointCloud\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bfd3e7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets visualize the image.\n",
    "img = sensors.images[CameraChannel.CAM_F0]\n",
    "plt.imshow(img.as_numpy)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad704424",
   "metadata": {},
   "outputs": [],
   "source": [
    "# And then the visualize pointcloud.\n",
    "pc = sensors.pointcloud[LidarChannel.MERGED_PC]\n",
    "plt.imshow(pc.render_image())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0123b5a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# All camera channels\n",
    "sensors = scenario.get_sensors_at_iteration(0, [channel for channel in CameraChannel])\n",
    "\n",
    "\n",
    "# Visualizing all 8 channels\n",
    "fig, axarr = plt.subplots(4, 2, figsize=(10, 15))\n",
    "channel_idx = 0\n",
    "for channel, img in sensors.images.items():\n",
    "    row, col = channel_idx % 4, channel_idx // 4\n",
    "    axarr[row][col].imshow(img.as_numpy)\n",
    "    axarr[row][col].title.set_text(channel.name)\n",
    "    channel_idx += 1"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "5aea00fa936a67f14323fa2d54163d7dc1f328c617e433b03a680f73ee2dd426"
  },
  "kernelspec": {
   "display_name": "",
   "language": "python",
   "name": ""
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
